Document missing event.ingested field on OTel-native data streams - #660
Open
jamesagarside wants to merge 1 commit into
Open
Document missing event.ingested field on OTel-native data streams#660jamesagarside wants to merge 1 commit into
jamesagarside wants to merge 1 commit into
Conversation
Contributor
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
alexandra5000
requested changes
Aug 14, 2026
Contributor
There was a problem hiding this comment.
Thanks a lot @jamesagarside! I would suggest moving this content to limitations.md (like you yourself suggested in the PR description). I would put it directly under/on the same heading level as: https://www.elastic.co/docs/reference/opentelemetry/compatibility/limitations#ingest-pipelines-and-dotted-field-names
As for your other notes, I'm afraid I can't 100% verify whether classic APM data streams get event.ingested. In any case, I think an SME should review this content as well. @felixbarny do you know the answers to James' doubts? Or could you tag someone who could review this from the technical point of view, please 🙏
Comment on lines
+168
to
+170
| :::{note} | ||
| Refer to [elastic/elasticsearch#100324](https://github.com/elastic/elasticsearch/issues/100324) for the open request to expose `event.ingested` as a data stream setting. | ||
| ::: |
Contributor
There was a problem hiding this comment.
Suggested change
| :::{note} | |
| Refer to [elastic/elasticsearch#100324](https://github.com/elastic/elasticsearch/issues/100324) for the open request to expose `event.ingested` as a data stream setting. | |
| ::: |
I don't think we need to link to the issue. If there is any critical information there, it should be available directly in the docs.
|
|
||
| ### Impact on detection rules | ||
|
|
||
| Elastic Security prebuilt detection rules that use `event.ingested` as their timestamp override, such as the External Alerts rule, report a `partial failure` execution status when their index patterns match OTel-native data streams: |
Contributor
There was a problem hiding this comment.
Suggested change
| Elastic Security prebuilt detection rules that use `event.ingested` as their timestamp override, such as the External Alerts rule, report a `partial failure` execution status when their index patterns match OTel-native data streams: | |
| Elastic Security prebuilt detection rules that use `event.ingested` as their timestamp override, such as the External Alerts rule, report a `partial failure` execution status when their index patterns match OTel-native data streams. For example: |
Comment on lines
+182
to
+227
| ### Populate `event.ingested` on OTel log data streams | ||
|
|
||
| `logs-otel@template` composes `logs@settings`, which sets `index.default_pipeline` to `logs@default-pipeline`. That pipeline calls the `logs@custom` ingest pipeline if it exists, which gives you an upgrade-safe extension point. | ||
|
|
||
| Create `logs@custom` to route OTel datasets to a dedicated pipeline: | ||
|
|
||
| ```console | ||
| PUT _ingest/pipeline/logs@custom | ||
| { | ||
| "processors": [ | ||
| { | ||
| "pipeline": { | ||
| "name": "logs-otel@custom", | ||
| "ignore_missing_pipeline": true, | ||
| "if": "$('data_stream.dataset', 'null').endsWith('.otel')" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Then create `logs-otel@custom` to set the field: | ||
|
|
||
| ```console | ||
| PUT _ingest/pipeline/logs-otel@custom | ||
| { | ||
| "field_access_pattern": "flexible", | ||
| "processors": [ | ||
| { | ||
| "set": { | ||
| "field": "attributes.event.ingested", | ||
| "value": "{{{_ingest.timestamp}}}", | ||
| "override": false | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Keep the following in mind: | ||
|
|
||
| * Set the field as `attributes.event.ingested`. Because `attributes` is a `passthrough` object in `logs-otel@mappings`, the field is then queryable under the bare name `event.ingested`. | ||
| * {applies_to}`stack: ga 9.2+` {applies_to}`serverless: ga` `field_access_pattern` must be `flexible` to write a dotted field name into a `passthrough` object. Refer to [field access pattern](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#access-source-pattern-flexible). | ||
| * You don't need to declare the field mapping. `ecs@mappings`, which `logs-otel@template` composes, has an `ecs_date` dynamic template matching `*.ingested`, so the field is mapped as `date`. | ||
|
|
||
| `metrics-otel@template` composes `metrics@tsdb-settings`, which doesn't set a `default_pipeline`, so there's no equivalent extension point for `metrics-*.otel-*` data streams. |
Contributor
There was a problem hiding this comment.
I would restructure this section as follows:
### Populate `event.ingested` on OTel log data streams
`logs-otel@template` composes `logs@settings`, which sets `index.default_pipeline` to `logs@default-pipeline`. That pipeline calls the `logs@custom` ingest pipeline if it exists, providing an upgrade-safe extension point.
1. Create `logs@custom` to route OTel datasets to a dedicated pipeline:
```console
PUT _ingest/pipeline/logs@custom
{
"processors": [
{
"pipeline": {
"name": "logs-otel@custom",
"ignore_missing_pipeline": true,
"if": "$('data_stream.dataset', 'null').endsWith('.otel')"
}
}
]
}
```
2. Create `logs-otel@custom` to set `attributes.event.ingested`. Because `attributes` is a `passthrough` object in `logs-otel@mappings`, the field is then queryable under the bare name `event.ingested`:
```
PUT _ingest/pipeline/logs-otel@custom
{
"field_access_pattern": "flexible",
"processors": [
{
"set": {
"field": "attributes.event.ingested",
"value": "{{{_ingest.timestamp}}}",
"override": false
}
}
]
}
```
:::{note}
{applies_to}`stack: ga 9.2+; serverless: ga`
Set `field_access_pattern` to [`flexible`](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines#flexible-field-access-pattern) to write a dotted field name into a `passthrough` object.
:::
:::{tip}
You don't need to declare the field mapping. `ecs@mappings`, which `logs-otel@template` composes, has an `ecs_date` dynamic template matching `*.ingested`, so the field is mapped as `date`.
:::
`metrics-otel@template` composes `metrics@tsdb-settings`, which doesn't set a `default_pipeline`, so there's no equivalent extension point for `metrics-*.otel-*` data streams.
Comment on lines
+229
to
+249
| ### Streams | ||
|
|
||
| The `logs@custom` pipeline doesn't apply to [Streams](docs-content://solutions/observability/streams/streams.md). {{kib}} generates its own index template for wired streams that composes only `<ancestor>@stream.layer` component templates and sets `default_pipeline` to `<stream>@stream.processing`, so `logs@settings` and `logs@custom` aren't part of the composition. | ||
|
|
||
| For a wired stream, add a [`set` processor](docs-content://solutions/observability/streams/processors/set.md) to a child stream. Root wired streams can't hold custom processing. | ||
|
|
||
| ```json | ||
| { | ||
| "action": "set", | ||
| "to": "attributes.event.ingested", | ||
| "copy_from": "_ingest.timestamp", | ||
| "override": false | ||
| } | ||
| ``` | ||
|
|
||
| Keep the following in mind: | ||
|
|
||
| * Use `copy_from`. The [Streamlang](docs-content://solutions/observability/streams/streamlang.md) `set` action rejects Mustache template syntax in `value`, and the `manual_ingest_pipeline` action isn't allowed in wired streams. | ||
| * The `to` value must be prefixed with `attributes.`. The bare field name is rejected. | ||
| * Wired streams are `dynamic: false`, so you must also [declare](docs-content://solutions/observability/streams/map-fields.md) `event.ingested` as a `date` field on the stream. Otherwise the value is stored but not indexed. | ||
|
|
Contributor
There was a problem hiding this comment.
For this section, I would also suggest a slight rewrite:
### Populate `event.ingested` in Streams
The `logs@custom` pipeline doesn't apply to [Streams](docs-content://solutions/observability/streams/streams.md). {{kib}} generates its own index template for wired streams that composes only `<ancestor>@stream.layer` component templates and sets `default_pipeline` to `<stream>@stream.processing`, so `logs@settings` and `logs@custom` aren't part of the composition.
Root wired streams can't hold custom processing, so add the processing to a child stream instead.
1. In the child stream, add a [`set` processor](docs-content://solutions/observability/streams/processors/set.md) targeting `attributes.event.ingested`:
```json
{
"action": "set",
"to": "attributes.event.ingested",
"copy_from": "_ingest.timestamp",
"override": false
}
```
:::{note}
Use `copy_from` rather than `value`. The [Streamlang](docs-content://solutions/observability/streams/streamlang.md) `set` action rejects Mustache template syntax in `value`, and the `manual_ingest_pipeline` action isn't allowed in wired streams.
:::
2. [Declare](docs-content://solutions/observability/streams/map-fields.md) `event.ingested` as a `date` field on the stream. Wired streams are `dynamic: false`, so the value is stored but not indexed until you declare the field.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this documents
event.ingestedis not populated on OTel-native data streams (logs-*.otel-*,metrics-*.otel-*), and I couldn't find this stated anywhere in the docs. It surfaces as a confusing partial failure in Elastic Security prebuilt detection rules that useevent.ingestedas their timestamp override.This adds a section to the data streams comparison page covering:
event.ingestedis set by.fleet_final_pipeline-1, which Fleet attaches asindex.final_pipelineonly to the data streams it manages.logs-otel@templateandmetrics-otel@templatecompose nofinal_pipeline, so nothing sets the field.partial failurewithThe following indices are missing the timestamp override field "event.ingested". The rule still runs and falls back to@timestamp, but loses the ingest-lag protection the override provides.logs@customingest pipeline extension point, writing toattributes.event.ingestedwithfield_access_pattern: flexible.logs@customdoes not apply to wired Streams, where the only route is acopy_fromprocessing step on a child stream, plus an explicit field declaration.I also note that
metrics-otel@templatecomposesmetrics@tsdb-settingsand has nodefault_pipeline, so there is no equivalent hook for OTel metrics data streams.Basis
This is based on behaviour I observed and verified first-hand on an Elastic Cloud Serverless Security project, Elasticsearch/Kibana 9.6. I've deliberately not claimed scope beyond that. I cross-checked the mechanism against the template and pipeline definitions in
elastic/elasticsearchmain:logs@default-pipeline.json— two processors: set@timestamp, then thelogs@customhook. Noevent.ingested.logs@settings.json— setsindex.default_pipelineonly, nofinal_pipeline.logs-otel@template.yaml/metrics-otel@template.yaml—composed_oflists, neither pulling in afinal_pipeline.ecs@mappings.json—ecs_datedynamic template matches*.ingested, hence no explicit mapping needed on classic OTel log data streams.On the workaround, I confirmed on the live cluster that after applying it the detection rule returned to
succeededacross multiple executions, that the field is queryable under the bare nameevent.ingestedand reported as such by_field_caps, and thatfield_access_pattern: flexibleis required to write the dotted path into theattributespassthrough object. For wired streams I confirmed that Streamlang'ssetrejects Mustache syntax invalue, thatmanual_ingest_pipelineis rejected in wired streams, and that_field_capsstayed empty until the field was declared on the stream.Related public issue, linked in the page: elastic/elasticsearch#100324.
Notes for reviewers
event.ingested.logs@customworkaround is something Elastic wants to recommend versus just documenting the gap.